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INTEGRABLE WAVELET TRANSFORMS WITH ABELIAN DILATION 

GROUPS 

B. CURREY, H. FUHR, K. TAYLOR 


Abstract. We consider a class of semidirect products G = K" xi H, with H a suitably 
chosen abelian matrix group. The choice of H ensures that there is a wavelet inversion 
formula, and we are looking for criteria to decide under which conditions there exists a 
wavelet such that the associated reproducing kernel is integrable. 

It is well-known that the existence of integrable wavelet coefficients is related to the 
question whether the unitary dual of G contains open compact sets. Our main general 
result reduces the latter problem to that of identifying compact open sets in the quotient 
space of all orbits of maximal dimension under the dual action of H on K”. This result 
is applied to study integrability for certain families of dilation groups; in particular, we 
give a characterization valid for connected abelian matrix groups acting in dimension 
three. 


1. Introduction 

Before we describe the aims of this paper, we need to introduce some terminology and 
notations. Let H < GL(?t,, M) denote a closed matrix group. The affine group generated 
by H and all translations is the semidirect product G = xi H. Elements of G are 
denoted by pairs {x,h) G R”' x R, with group law {x,h){y,g) = (x + hy,hg). G acts 
unitarily on L^(R’^) via the quasiregular representation vr, dehned by 

'^{x,h)f{y) = \det{h)\-^f{h-\y - x)) . 

Given rj G L^(G), the associated continuous wavelet transform is the mapping 

V, : L2(R") ^ G,{G) , U,/(x, h) = if, n{x, h)y) . 

Here Gb{G) denotes the space of bounded continuous functions on G. Whenever U? is an 
isometry into L^(G) (dehned with respect to left Haar measure), we call rj admissible. In 
turn, the group H will be called admissible if there exists an admissible vector for vr. 

Admissible vectors and groups have been studied extensively in the past twenty years, 
with varying degrees of generality. A small and subjective sample of the literature is 
[21, 3, 10, 18, 11, 12]. 
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The property we intend to investigate in this paper is integrability of the quasi-regular 
representation. We call a representation integrable if there is an admissible vector 77 
such that A(j^^‘^VrjT] is in L^(G). By slight abuse of notation we will call such vectors 
g integrable admissible vectors. A major motivation for studying integrability in this 
context is the coorbit theory due to Feichtinger and Grochenig [ 8 , 13], which heavily 
relies on integrability. Coorbit spaces associated to a quasi-regular representation can be 
interpreted as a family of smoothness spaces induced by the action of the group. While 
the initial dehnition also relied on irreducibility of the representation, recent work (e.g. 
[ 6 ]) suggests that a meaningful theory can also be developed for reducible representations, 
as long as integrability (or somewhat weaker conditions) are retained. 

Besides these connections to function spaces, integrability is also a representation-theoretic 
property of independent interest. For any admissible vector p, isometry of Vj, implies that 
VjjP = Vr,ri = Vr^rj*Vj^rj. Here we used the notation f{x) = f{x~^) for any function / on G. 
If, in addition, F = G L^(G), this implies that F is a projection in the group al¬ 

gebra L^(G), i.e. it fulhlls F = F*F = F*, where F*{x) = Ag{x)~^F{x~^). The question 
which group algebras contain projections has been studied, e.g., in [2, 14, 15, 16, 24, 25]. 
A necessary representation-theoretic condition for the existence of projections in hails 
from the following observation: If F G L^(G) is a projection, then supp(F) C G is a 
compact open set. (Note that here, as in the following, our dehnition of compactness 
does not include the Hausdorff property.) Here F is the group Fourier transform of F, 
obtained by integrating F against irreducible representations, and G denotes the unitary 
dual of G, i.e. the equivalence classes of irreducible representations of G. The topology on 
G is the Fell topology, dehned in terms of uniform convergence of matrix coefficients on 
compact subsets of G, see [7, 9] for details. Thus, a necessary (but usually not sufficient) 
condition for the existence of a projection in is the existence of a compact open set in 
the dual. 

For the setting studied here, the most signihcant precursors are [14, 16]; for a unihed 
account of the results see also [17]. The paper [14] contains a full characterization of 
integrability for the case H = exp(RA) (unlike our paper, [14] does not impose any 
condition on the spectrum of A). Briehy, the quasiregular representation associated to 
such a group is integrable iff the group is strictly contractive; or more technically, iff the 
signs of ReA coincide, for all eigenvalues A of A. 

In the following, we shall mostly concentrate on the following class of matrix groups: F[ 
is an abelian connected matrix group satisfying spec(/i) C M’*', for all h E H. The chief 
reason for focussing on this class is that there exist easily checked admissibility criteria 
for it: H is admissible iff det|iif ^ 1, and in addition, the dual action of F[ is free almost 
everywhere, see [4, Theorem 23]. Here, the dual action is dehned by {h,x) 1 —)■ h~'^x, 
where h~'^ = ■ We let d = dim(Fr). 

The following theorem is the chief general result of this paper. Before we formulate it, 
we need some additional notation. Orbits under are submanifolds of R”. If a matrix 
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group H is fixed, we define, for i G N: 

= {t G : dim{H^x) = i} . 

Clearly, Oi = ^ whenever i > d. For the class of groups that we consider here, the dual 
action is free almost everywhere iff Od 7 ^ 0- The “only if’-part is clear since the dimension 
of every free orbit equals the group dimension. For the “if’-part, note that if Od is 
nonempty, then it is Zariski-open, and its complement has measure zero. Furthermore, 
the action of H on Od is free: Since group dimension and orbit dimension coincide, the 
hxed groups associated to Od are discrete. But they are also connected, hence trivial. 

Finally, we endow the orbit space with a pseudo-image of Lebesgue measure. 

Theorem 1. Let H he a d-dimensional connected, admissible abelian matrix group sat¬ 
isfying spec(h) C M"*" for all h E H. If n is integrable, there exists a compact conull set 
C C Od/H^. 

The necessary compactness condition simplihes considerably, if more is known about the 
orbit space. 

Corollary 2. Let H be a d-dimensional connected, admissible abelian matrix group sat¬ 
isfying spec(h) C M"*" for all h E H. Assume in addition that Od/H^ is Hausdorff. Then, 
if n is integrable, Od/H^ is compact. 

Proof. If OdlH^ is Hausdorff, then every compact open subset C is closed as well, and 
thus it is a union of connected components of Od/H"^■ All connected components of Od 
are open, thus of positive Lebesgue measure. Hence all compact components of Od/H'^ 
have positive measure, as well. A compact conull subset C, if it exists, therefore must be 
all of Od/H^. □ 

A subset C C is called a topological section for an iL^-invariant subset H C if 
the map H x C ^ U,{h,c) ^ h'^c is a homeomorphism. Note that the existence of a 
topological section implies that H acts freely on U. Furthermore, the natural bijection 
C ^ U/ is a homeomorphism as well, implying in particular that the quotient space 
is Hausdorff. 

Corollary 3. Let H be a d-dimensional connected, admissible abelian matrix group sat¬ 
isfying spec(/i) C for all h E H. If there exists a topological section C <Z Od for the 
dual action, then tt is integrable iff C is compact. 

Proof. If C is a topological section, then the canonical bijection C -E Od/H"^ is a 
homeomorphism. In particular, the latter space is Hausdorff, and it is compact iff C 
is compact. Thus one direction is immediate from the previous corollary. The other 
direction follows from [16, Proposition 3.1]. □ 

2. Topological properties of dual orbit spaces and integrability 

In this subsection we collect some general observations concerning dual orbit spaces. We 
hrst recall an observation from [16]: 
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Lemma 4. Let U C ML be arbitrary. Then U/is compact iff there exists a compact 
set C <Z U with H^C = U. 

Proof. Compare the proof of [16, Corollary 4.2], □ 

We further note that, as a quotient space of a second countable space, is second 

countable as well, and so are all its subspaces. It follows that a subset is compact iff it 
is sequentially compact. Our discussion of concrete cases in the following sections will 
mostly rely on this observation. 

Our next aim is to generalize a known sufficient condition for the existence of integrable 
admissible vectors. It has been demonstrated in [16, Proposition 3.1] how to construct, 
given an open conull subset U having a topological section with compact closure inside 
U, admissible vectors that give rise to an integrable projection. Our aim is to replace the 
topological section by a weaker notion, a topological quasi-section. For this purpose, we 
borrow some notation introduced by Richard Palais [22]: Given Y, Z (Z M”, we let 

{(Y, Z)) = {h e H : h^Y n Z (f} . 

Definition 5. Let U C M” be open and iL^-invariant. A subset C G U is called 
topological quasi-section (for U) if it is open, with H^C = U, and such that ((G, G)) is 
relatively compact. □ 

The following lemma is clear, but useful to note. 

Lemma 6. Let U C M^ be open and H'^-invariant. Assume that C G U is a topological 
section. Let V G H be a relatively compact open neighborhood of the identity. Then 
VC G Lf is a topological quasi-section. 

We already noted that the existence of a topological section implies that the action is free. 
In the case of a compact quasi-section, a weakened version of this observation applies 

Lemma 7. Let U G be open and H'^-invariant. If U admits a topological quasi¬ 
section, then is compact for all f eU. 

Proof. If G is a topological quasi-section, it is sufficient to prove compactness of for 

all f & C. But this follows from the fact that ((G, G)) contains the stabilizer. □ 

Lemma 8. Let U G M^ be open and H'^-invariant. If there exists a topological quasi¬ 
section in U, then for all compact sets Ci,C 2 G U, ((Gi,G 2 )) is relatively compact. 

Proof. Denote the quasi-section by G. Then, by compactness of the Gj, we have Gj C 
FjG, for suitable hnite Fj C iL (f = 1, 2). A straightforward computation shows that 
((Gi,G2)) C F2{{C,C))Ff^ , which is relatively compact. □ 

We are particularly interested in compact orbit spaces. In this setting, the notion of a 
quasi-section becomes particularly useful. In order to properly appreciate the usefulness 
of the following quite simple result, recall that the computation of cross-sections can be 
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quite cumbersome. What is worse, the failure to produce a topological section usually 
does not provide any clue regarding the existence of snch a section. 

By contrast, as the following proposition shows, there is no “clever” way of picking quasi¬ 
sections, when the orbit space is compact: Either any relatively compact open set meeting 
every orbit will do, or none. Since topological sections give rise to qnasi-sections, this 
provides a usefnl necessary condition for existence of topological sections. 

Proposition 9. Let U <ZW^ be open and H'^-invariant. Assume that U / is compact, 
and let C <Z U denote a relatively compact open set with H^C = U. Then U has a 
topological quasi-section iff C is a topological quasi-section, i.e., iff {{C,C)) is relatively 
compact. 

In particular, if ((C, C)) is not relatively compact, then U does not have a topological 
section. 

Proof. The “if’-direction is clear. The other direction follows from Lemma 8. 

The statement concerning topological sections now follows from Lemma 6. □ 

The following result provides the extension of [16, Proposition 3.1]. 

Proposition 10. Let H < GL(n, M) be a closed matrix group. Suppose there exists an 
open H'^-invariant set U C that admits a relatively compact topological quasi-section 
for the dual action. Then the restriction nu of the quasi-regular representation to the 
invariant closed subspace Hu = {/ ^ L^(G) : f ■ lu = f} is integrable. 

Proof. We adapt the proof from [16]. Let C <Z U denote the relatively compact qnasi- 
section. Pick a relatively compact open set W with C (Z W C U. Let 0 G Cff{U) with 
Ic — 0 — Iw- Foi’ i , dehne 

^( 0 = / \<f{h^0?dh. 

J H 

Then u is a well-dehned smooth function on Lf\ Let G 17 be arbitrary, and let U' Z U 
denote a compact neighborhood of Then for all G U', 

a(0= [ _ m^a^dh. 

and Lemma 8 ensnres that the integration domain is compact. In particular, a is well- 
dehned, and compactness of the integration domain allows to interchange differentiation 
and integration, which shows that a is smooth. 

Fnrthermore, for every ^ E U, the map h eA f) is a positive continnous fnnction, and 
not identically zero: There exists h E H with hf ^ E C, and thus \(j){hFf)\'^ > 1 by choice 
of (j). Hence > 0. 

But then ^ is a smooth, compactly supported function on U, which we extend 

trivially to all of M”. We let g E L^(M”') denotes its inverse Fonrier transform; by 
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construction, g is in fact a Schwartz function. Letting Qh '■ i ^ that 

'^h = 0 whenever h ^ Cw,vl, and Lemma 8 implies that this set is relatively compact. 

Using the properties of 'g collected thus far, the proof of [16, Proposition 3.1] is seen to 
go through. Since our notation somewhat differs from that in [16], we include a short 
sketch. We hrst rewrite the wavelet transform as 

Vgg{x,h) = |det(/i)|^/^ ( t ) . 

It follows that we can estimate 

\\^G^'‘^yg9\W < ! |det(/i)|"^/^A"^/^(/i)d/i . 

JH ^ ^ Li 

The vector-valued mapping 

h ^ g ■ gh ^ C^{U) 

is continuous with respect to the Schwartz topology, implying that the mapping 

h H- ||(^- det(/i)|"^/^A“^'^^(/i) 

is continuous as well, and we noted above that it is also compactly supported. This 
implies G L^(G). □ 

_1 /o 

Note that the proof shows that we may replace the factor A^ ' by any weight function 
on H that is bounded on compact sets. 

Unlike for topological sections, the existence of quasi-sect ions does not imply that the 
group action is free. In particular, we can apply the proposition to the case where U is 
an open dual orbit with compact stabilizer. The subrepresentations ttu corresponding to 
such orbits are precisely the discrete series subrepresentations [12, Corollary 21]. Hence 
we obtain the following result. 

Corollary 11. Let H < GL(n, M) be a closed matrix group. Every discrete series sub¬ 
representation of the quasi-regular representation n is integrable. 

While the notion of quasi-section may seem somewhat ad hoc, a result by Palais shows 
that the existence of such a quasi-section naturally connects to well-studied topological 
properties of the orbit space. Recall for the following that properness of the dual action on 
an i/^-invariant subset U means that the inverse image of any compact subset C G U xU 
under the mapping H x U ^ U x U, {h, f) is compact. 

Proposition 12. Let U G ML be open and H'^-invariant. Then the following statements 
are equivalent: 

(a) There exists a topological quasi-section C G U such that C is contained in U and 
compact. 

(b) The orbit space U/H^ is compact and the dual action of H on U is proper. 

If any of these conditions hold, Lf/H^ is Hausdorff and metrizable. 
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Proof. First assume that (a) holds. Then U/H"^ is clearly compact. We hrst note that 
by Lemma 8, we have that ((Ci, C 2 )) is relatively compact, for all open relatively compact 
subsets Ci,C 2 of U. In the parlance of [22], this means that Ci,C 2 are relatively thin, 
and since this holds for all open, relatively compact neighborhoods of arbitrary points, it 
follows that each point in U has a small neighborhood (again in the terminology of [22]). 
Now implication (1) ^ (3) of [22, Theorem 1.2.9] yields that the dual action on U is 
proper, and we have concluded (6). 

Conversely, assume that (6) holds. Pick a compact set C* C 17 with H^C = U. Pick 
VF D (7 open, with closure compactly contained in U. Then implication (3) ^ (4) of 
[22, Theorem 1.2.9] implies that {{W,W)) is relatively compact, i.e., hF is a topological 
quasi-section. 

The Hausdorff property of U /follows by [22, Theorem 1.2.9], (3) ^ (2), and we 
already noted that 17/77^ is second countable. Metrizability now follows from Urysohn’s 
metrization theorem. □ 

Remark 13. The results of this section provide a partial converse to Theorem 1: Let 
77 be a <7-dimensional admissible abelian matrix group, and assume that there exists an 
open conull 77-invariant set U <Z Od such that 17/77^ is compact. If in addition the dual 
action of 77 on 7/ is proper, then Propositions 10 and 12 imply that tt is integrable. 

It is currently unknown whether the gap between this statement and Theorem 1 can 
actually be closed: Proposition 18 (b) below presents a matrix group 77 for which the 
dual action has a conull open compact subset of the orbit space, but the properness 
condition is violated. For this example, the question of integrability of the associated 
quasi-regular representation is currently open. □ 


3. Proof of Theorem 1 

Throughout this section, 77 denotes an abelian matrix group fulhlling the conditions of 
Theorem 1. In particular, the dual action of 77 is free almost everywhere, and thus 
d = dim(77) < n. Furthermore, 77 is connected and all elements h E H have positive 
spectrum. 

Let us hrst comment on the proof strategy: The chief representation-theoretic tool em¬ 
ployed in [14, 16] was the Mackey machine. However, while there are useful results pro¬ 
viding topological information (such as continuity of induction), in general the Mackey 
machine only yields a full description of the dual as a set, and comprehensive information 
on the topological structure is not easily obtained this way. 

For this reason we choose an alternative approach, which is available because the semidi- 
rect products under consideration turn out to be exponential. In fact, G is a completely 
solvable Lie group (by Lemma 14 below). Thus G is homeomorphic to the coadjoint orbit 
space of G: Denote by g the Lie algebra of G, and by g* its dual. G acts on g* by the 
coadjoint representation Ad*, and there exists a canonical bijection k : g*/Ad*(G) —)■ G, 
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the Kirillov-Bernat correspondence. Furthermore, the bijection is in fact a homeomor- 
phism, when we endow the orbit space with the quotient topology; see [19] for details. 
This observation allows one to employ the methods developed by Currey and Penney 
to derive sufficient information on the coadjoint orbit space for the proof of Theorem 1. 
A further important ingredient of the proof is the close relationship between dual and 
coadjoint orbits. 


3.1. Adjoint and coadjoint representation of G. We identify elements of the Lie 

algebra g with pairs {v,X)'^ G x [). Accordingly, g* is canonically isomorphic to 
M” X 1)*; i.e. we have 

((e,W),(a;,X)) = (e,a;) + (W,X) . 


In order to use block matrix calculus, we will write elements of the product spaces as 


column vectors, i.e. 


w 

X 


X 1), 


X* 


X [)*, etc. 


The following formulae are conveniently verihed by realizing that the mapping [x, h) i—)■ 

is an isomorphism onto a closed group of (n + 1) x (n + l)-matrices. Let 

Ad : G GL(g) and Ad* : G —)■ GL( 0 *) denote the adjoint and coadjoint representations 
of the group. Let ad and ad* denote their Lie algebra counterparts. 


f h X 

VO 1 


We then obtain the following formulae, using that the adjoint representation of H is 
trivial: 

(1) Ad(i, A) ( y ) = (-Ad(A)(K) . I + hw, M(h)Yf = ([ * ]) ■ ([“]), 

with 99a, : [) —)■ R”, Y i—)■ —Y ■ x. For the coadjoint representation, we obtain 

(2) Ad*{x,h) ^ 1 ) ■ ( W ) ’ 

with 99 * : R” ^ [)* the map dual to (py. 

We next turn to the Lie algebra representations. Here we obtain 

( 3 ) {y) • 

and finally 

(4) ad {x,Z) 0 ) ■ ( F* ) • 

From these formulae, we can derive a number of simple but important observations. 
Lemma 14. G is completely solvable. 


Proof. By Lemma 3, we hnd spec{ad{x, Z)) C {0} U spec(F). But the assumption 
that spec(RZ) C R^ implies that spec(F) C R, and thus spec{ad{x, Z)) C R, for all 
{x, Z) G 0 . But this is equivalent to complete solvability of G. □ 
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Recall that the Kirillov-Bernat correspondence k is characterized as follows. For Y*) G 
0 *, suppose that p is a subalgebra of g such that (a) [p,p] C ker(,^,’F*), and (b) p is a 
maximal subalgebra with respect to the property (a). Then one has 

(5) dim Ad*(G)(.^, y*) = 2 dim( 0 /p). 

For each Y*) e g*, there is a subalgebra p satisfying (a), (b), and also (c) y*)+p-‘- C 

Ad*(G')(^,y*). (Here p^ = {(C,^*) G g* : p C ker(C,X*)}. ) Then (^,y*) dehnes a 
unitary character y = X{^,y*) on P = exp p by 

x(exp(a;,X)) = 

and the unitary representation 7 r((.^, F*), p) ;= indp(x) is irreducible. The unitary equi¬ 
valence class [ 7 r((.^, y*), p)] of 7 r((.^, F*), p) is independent of the choice of p, as well as 
independent of the choice of G Ad*(G)(.^, F*), and one has /t(Ad*(G)(^, F*)) = 

H{^,Y*),p)]. 

The following lemma clarihes the relationship between coadjoint orbits, dual orbits, and 
the dual G, which is particularly transparent for the class of groups that we study here. 
For the more general setting, the interested reader is referred to [1]. The formulation of 
the lemma requires two more pieces of notation: We let = {X G p : = 0}; i.e., p^ 

is the Lie algebra of the dual stabilizer Its annihilator is given by p^ = {F* G p* ; 
(F*,P^) = {0}}. 

Lemma 15. (a) For all Y*) G g* we have 

(6) Ad*(G)«, Y‘) = (ffV) X (r- + bU 

(b) The projection map P : g* ^ MT intertwines the coadjoint and dual actions. It 

therefore induces a continuous map P ; g*/Ad*(G) —)■ /H'^. 

(c) Let O* C g* denote the union of orbits of dimensions i. Then P : O^d/^d*{G) —)■ 
Od/H^ is a homeomorphism. 

(d) Let m : Od/H^ G be the Mackey injection defined by = [indRn(x^)]. 

Then the restriction of m o P to O^d!^d*{G) coincides with the restriction of k, 
to 02 *./Ad*(G) 

Proof. To verify ( 6 ), put p(^,y*) = M” x p^; using (2) we see that P(^,y*) is a subalgebra 
satisfying conditions (a), (b), and (c) with respect to (^,F*). Hence 

dim Ad*(G')(.^, F*) = 2 dimg/pg = 2 dimp/p^ = 2 dimiL^,^. 

A direct calculation again using (2) shows that 

Ad*(G)({,r-)c(i/T)x(y + i,p, 

and it is clear that dim {H^^) x {Y* + p^) = 2dimiL^,^, so ( 6 ) follows. 

For part (c), we have seen that the coadjoint orbit dimension of (.^, F*) is twice the dual 
orbit dimension of f. In particular, (^, F*) G O^d iff ■C ^ ^d- Furthermore, H acts freely 
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on Od, which for ^ e Oa implies = [)*, and thus Ad*(G)(.^, F*) = ^ Hence 

P is bijective on 02d/Ad*(G), and thus a homeomorphism. 

Finally, the above shows that for Y*) G subalgebra p = M” satisfies conditions 

(a), (b) and (c) for the Kirillov-Bernat correspondence. The characters on M” defined by 
{^,Y*) and ^ are obviously the same, giving the equality of the induced representations 
7r((^,F*),p) = ind^„(x^) so 

A(Ad*(G)(e,F*)) = [7r((e,F*),p)] = [indg.(x^)] =m(if-0. 

□ 


3.2. A layering of p*. The reduction to dual orbits of maximal dimension will rely on a 
corresponding result for the space of coadjoint orbits. The following lemma contains the 
vital parts of the proof. For the most part, it is a rephrasing of the main result of [5], 
adapted to the our specific needs. 

Lemma 16. There exists a decomposition of 0 * into Ad* {G)-invariant, disjoint sets 
p '^he following properties: 

(i) For all 1 < k < p: open. 

(ii) For all I < j < p: All coadjoint orbits in have the same dimension dj. The 
sequence di,... ,dp is nonincreasing. 

(iii) For all I < j < p: H*/Ad*(G) is Hausdorff. 

(iv) For all 1 < j < p we have that Tl* = P~^{P{Q*)). 

(v) Whenever dj < 2d, all connected components of Of are noncompact. 

Proof. The proof relies on the explicit construction of cross-sections for the coadjoint 
orbits of G, as described in [5]. The starting point for this construction is a Jordan-Holder 
basis of 0 . In our setting, this can be obtained with particular ease: By [4, Theorem 4], 
there exists a basis of M” with respect to which all elements of H are upper triangular. 
We denote this basis by Xi,..., X„, and complement it by a basis Xn+i, ..., Xn+d of i). 
It is then straightforward to check that 

Qj = span(Xi : i <j) , 

defines an increasing sequence of ideals 0 ^ in 0 . Hence Xi,..., Xd+n is the desired Jordan- 
Holder basis. 

Consequently, the dual basis X^,... C 0 * is a Jordan-Holder basis for the dual, 

with the usual, implicit convention that this time, the subspaces 

Vj = span(X; :i>j) 

are submodules. It follows that G acts on Q*/Vj, and the construction of the layering will 
be obtained from data associated to the actions of G on the various quotient spaces. 

Given (,^, Y*)"^ G 0 * and 1 < j < n J- d, we let 

L,(e, W) = {KX) G 0 ; ad*KX)(e, Wf G V,} . 
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By definition, Lj{^,Y*) is therefore the Lie algebra of the hxed group of + Vj 

with respect to the action of G. 

Now, given Lj{^,Y*), for j = 1,... ,n + d, R. Penney and the hrst author describe in 
[5] how to associate an index set j(.^, Y*) and an ordering scheme q;(.^, Y*) for j(,^, Y*), in 
such a way that the layering is obtained by picking suitable tuples j, a and letting 

(7) = {(e, 7^*) : i(e, 7^*) = i, «(e, 7^*) = «} . 

Here is the set of jump indices for (,^,y*), indicating when the dimension of 

Ad*(G)(.^, y*) + Vj_|_i exceeds that of Ad*(G)(.^, F*) + Vj. Hence, the cardinality of the 
jump set is equal to orbit dimension. 

In addition, one can number the nonempty layers as fli,..., flp in such a way that prop¬ 
erty (i) is guaranteed (see [5, Main Theorem (c)]). In each layer, the dimensions in the 
coadjoint orbits is constant (see [5, Main Theorem (b)]). The property that the orbit 
dimension is nondecreasing follows since the layering is a rehnement of the layering de¬ 
scribed in [23], as pointed out on [5, p. 311]. One can check that the coarser layering has 
this property by consulting the dehnition of the total ordering given directly after [23, 
Corollary 1.1.2]. 

Furthermore, there exists for each layer Oj a cross-section Sj C O* and a diffeomorphism 
0j : Sj X IFj —)■ O (with a suitably chosen vector space Wi) such that for each (,^, F*) G S 
we have 0j((.^, F*), hR) = Ad*(G)(.^, F*) [5, Main Theorem (f)]. But this easily implies 
that f2j/Ad*(G) is homeomorphic to Ej, in particular, it is Hausdorff. This takes care of 
{in). 

For the proof of part (iv) we observe that, since ad*(a;, F)(.^*, F*) = ad*(a;, F)(.^*, 0) 
(see (4)), it follows that Lj{^,Y*) = Lj{^,0), for all 1 < j < n + d. But this implies 
j(.^, F*) = j(,^, 0), and the same for a, and (iv) follows. 

Now (v) is an easy consequence of (iv) and of ( 6 ): Assume that (.^,F*)'^ G flj, and 
dj < 2d. Then dj = 2k with k < d, and thus codim(I)^) > 1. Let V C i)* denote a 
complement of 1)^. Then the map 

F 3 F* ^ {H^O X {Z* + 1)^) 

dehnes a homeomorphism onto the closed subset P ^{{H^^}) of fl*/Ad*(G). This set 
contains the coadjoint orbit of (.^, F*)^, and it is connected and noncompact. Hence the 
connected component in f2*/Ad*(G) containing that orbit is not compact. □ 

The properties of the lemma will allow us to “peel off’ all layers in the coadjoint orbit 
space that consist of orbits of less than maximal dimension. 

Theorem 17. Let C C g*/Ad*(G) denote an open compact set. Then C C 

Proof. Let denote the layering from Lemma 16. Note that Q* consists of 

the zero-dimensional orbits. Suppose that C C 0 */Ad*(G) is compact and open. Then 
Cp = C 0 r2*/Ad*(G) is a compact subset of f2*/Ad*(G), which is Hausdorff by part (hi) 
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of Lemma 16. Furthermore, it is open, hence it contains every connected components 
of r2*/Ad*(G) it intersects. But these are noncompact, by Lemma 16.(v). Thus Cr, H 
fi;/Ad*(G) is empty. 

If dp-i < d, we can repeat the argument, using that C ljj<n-i closed, to obtain 
that C n r2*_]^/Ad*(G) = 0. Thus one concludes iteratively C fl r2*/Ad*(G) = 0, until 
dj = d. □ 


3.3. Proof of Theorem 1. Suppose that rj G L^(M'^) is an admissible vector with F = 
G L^(G). Denote by F the group Fourier transform. Then supp(F) C G is a 
compact open set. Let n : g*/Ad*(G) — )■ G denote the Kirillov-Bernat correspondence. 
It is a homeomorphism [19], and thus Theorem 17 implies that supp(F) = k{C) for some 
compact open subset C C 02^/k.d*{G)). By Lemma 15, P{G) is a compact open subset 
of Od/H^. 

Now let "H = Vp{L‘^{W^)) C L^(G). The measure class on the Borel space G for the 
quasiregular representation n is supported on O^d!^d*{G) = Od/H'^, and is the push 
forward of the Plancherel measure on M” (see for example [20, Theorem 7.1]). Hence the 
restriction of the regular representation of G to "H is quasiequivalent with the regular 
representation. Since convolution by F is projection of L^{G) onto "H, then the support 
of F must be conull. This entails that G is conull, and hence P{G) C must be conull 
as well. 


4. The 3x3 case. 

In this section we specialize to the case where n = 3, with the aim of proving a charac¬ 
terization. The proposition lists each connected abelian F[ < GL(3,M) for which M?/H'^ 
contains a compact open subset. Of particular interest is case (b) for a[5 ^ 0, providing 
an example of a conull compact open subset of the orbit space where Proposition 10 is 
not applicable. Even though this problem seems amenable to direct calculations, the 
question whether an integrable admissible vector exists for this case is currently open. 

Proposition 18. Let i) be a commutative Lie subalgebra o/gl(3,M) having dimension 
d>2, and suppose that there is a basis o/M^ for which one of the following cases holds. 

(a) 1)^ = where 



1 —a 


'o 

A = 

a 1 

, and B = 

0 


0 


1 


with a ^ 0. 
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(b) = {A, B)]^ where 



with at least one of a or (3 positive. 


and B 



(c) = {A,X,Z)m. where 



and Y 



(d) = {A,B,X)]s^ where 

"i 1 1 ' 

A = 1 , B = 0 , and X = 1 0 

oj L L 

(e) [)^ = {A, B, C)k where 

"i 1 1 ' 

A = 0 , B = 1 , and C = 0 

oJ L L ^ 

Then Od/H^ is compact. Otherwise, Od/H'^ contains no conull, compact open subset. 

In case {h), there exists a topological section for Od/H'^ iff otfl = 0, whereas for a/3 ^ 0 
there does not even exist a topological guasi-section. In all remaining cases, 02/11^ 
admits a topological section. 

If none of the above cases applies, then n is not integrable. Conversely, n is integrable in 
all listed cases possessing a topological section. 

Note that when d = 3, Od consists of open orbits, and it is easy to check directly that the 
groups generated in examples (c), (d), and (e) above generate hnitely many open orbits. 
We elaborate on this in the cases below, but the main focus of the proof will be on the 
two-dimensional case. 

We hrst decompose into generalized eigenspaces for t). Let A be a complex linear form 
on (), and put 

E\ = nAef,kerc3(A - X{A)lf 

where each A is extended to an endomorphism of as usual. If Ex ^ {0}, then we say 
that A is root for t). Since t) is commutative, there is a hnite set H of roots such that 
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Accordingly we write v = ■ If then A G 7^ also, and for v G 

yW = y(A)_ yy-g then form a set {Ai,..., Ap} by choosing exactly one member of each 
conjugate pair in TZ. If A is real, put Vj = Ex. fl and otherwise put Vj = Ex.. Then 
the map i —)■ Id © • • • © dehned by l{v) = is an R-linear isomorphism. 

The preceding generalized eigenspace decomposition can be carried out for any commuta¬ 
tive Lie algebra 1) in gl(n, M). We now make some observations particular to the situation 
at hand, where n = 3. First, note that if Xj is not real, then = C ~ and p = 2. 
Second, observe that if there is a non-zero nilpotent element in i), then p < 2 and all roots 
are real. Let n = fl^^^ker Aj denote the subspace of nilpotent elements of (). In light of 
these observations we itemize several cases. 

Case 0: p = 1 and Ai = 0. Here [) = n is a Lie subalgebra of the Heisenberg Lie algebra, 
and our assumption that t) is commutative then requires that d = 2 . 

Case 1: p = 1 and Ai ^ 0. Here the dimension of n is one or two, and Ai is real. Suppose 
that dim n = 2; then for any basis of there must be X G n for which 0:21 7 ^ 0, and 
then the assumption that t) is commutative implies that for all H G n, P 32 = 0. This is 
example (d). It is easily seen that in this case O 3 is the union of two open orbits. 

Case 2: p = 2 and Ai and A 2 are linearly dependent over M. Here one of the generalized 
eigenspaces has dimension one, and it follows that the dimension of n is one, so d = 2 . 

Case 3: p = 2 and Ai and A 2 are linearly independent over R. In this case, if n 7 ^ {0}, 
then both roots are real, and we are necessarily in example (d) above, whence O 3 is the 
union of four open orbits. Otherwise there can be a complex root. 

Case 4: p = 3. Here we must have n = {0}, and the dimension of the real span of the 
roots is d. If d = 3 then we clearly obtain example (e), where O 3 is a union of eight open 
orbits. Otherwise, with Ai and A 2 linearly independent, the question of compactness of 
02 /H^ will depend upon how A 3 is written as a linear combination of Ai and A 2 . 


The following lemma allows us to assume that all roots are non-vanishing, and in partic¬ 
ular, to eliminate Case 0. 


Lemma 19. Suppose that A = 0 for some \ E TZ. Then R^/id^ contains no non-empty 
compact open subset. 

Proof. We have a generalized eigenspace Vq in which the action of is unipotent. 
Choose a basis ci,..., for Vq for which every element of has a lower triangular 
matrix, and let U be the span of 62 ,... together with any other generalized eigenspaces 
(if p = 1 and ni = 1 then U = {0}). Then U is idWjnvariant. Let tt : R^ —)■ R^/H = Rci 
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be the canonical map. Then vr is continuous, open, and if ^-invariant, and so gives a 
well-dehned, continuous open map 7f : M^/if^ —)■ Mci. If C is a compact open subset of 
M^/if^, then Tf{C) is compact and open in 3fii, hence empty. Thus C is empty. 

□ 

Next we prove two lemmas for Case 1. When an ordered basis {ci, 62, 63} for is chosen, 
let pj : —)■ R be the linear (coordinate) function dual to e*. 


Lemma 20. Suppose that d = 2 and we are in Case 1, and write = (A, X)® where 
Ai(yl) = 1 and Ai(X) = 0. Choose a basis for R^ for which is lower triangular, and 


write 


X 


0 

X21 0 

T 31 a;32 0 


Put 


^2 = {v : P 2 (^v) 7 ^ 0 } 

and 

hl3 = {p : P2{Xv) = 0,p3{Xv) 7 ^ 0 }. 

Then the sets 172 and his are H'^-invariant and O2 = U ^3. When Qb is nonempty, 
then 

Sfe = {t G fife : Pbip) = 0 and pb{Xv) = ±1} 
is a topological section for the -orbits in fib- 


Proof. We have 172 7^ 0 if and only if 0:21 7^ 0. In this case p 2 {Xv) = 0:21^1, and so 
fi2 = {t : Ti 7^ 0}. Thus 172 is if-invariant with 

P2{XC\) = C^^^^^P2{Xv). 

Now suppose that Vi = 0. If p^lXv) 7^ 0 for some v then p = 1 (i.e., there is only one 
generalized Jordan block ) and p 3 {e^^Xv) = e^^^^^lp 3 {Xv) for all v. 


Observe that if fife is nonempty, then the restriction of the map 0 : fife ^ R* x R dehned 
by 

0(n) = (pfe(Xn),'i;fe) 

to each if-orbit has rank 2. Thus fife C 02- If n G £^2 then the action of is non-trivial 
and so n G fife, 6 = 2 or 3. 


To see that Sfe is a section, 
any i G R, 


let n G fife. Note that by dehnition of b, pb{X‘^v) 
Pbie^^v) = pb{v) + tpb{Xv). 


Put s = — In |pfe(Xn)|, and 


Pbjv) 

Pb{Xv)' 


0, so for 
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Put V* = Then pi,{Xv*) = pb{Xe^^v) = e^ph{Xv) = ±1, and 

Pb{v*) = pb{ v) + tpb{X v) = 0. 

Thus V* belongs to Sf,. On the other hand if v* E Sf, and v = E Sb also, then 

Pb(Xv) = pb(Xe^\*) = e^pb(Xv*) implies s = 0 and 

0 = pb(v) = Pb(e*^v) = tpb(Xv) 

shows that t = 0. Thus Eb meets each orbit at exactly one point. Furthermore, the 
explicit computations determining s, t and hnally v* from v show that the map v ^ v* 
is continuous. This implies that the section is topological. □ 

Lemma 21. Suppose that d = 2 and we are in Case 1 and that O 2 consists of only one 
layer: O 2 = fla = ^- Then 02lH^ does not contain a non-empty compact open subset. 

Proof. Let E be the topological section for O 2 constructed above, let a : 02 /H^ -E E 
be the natural homeomorphism, and let C* be a compact open subset of O 2 /H"’". Then 
(t(C') is a compact and open snbset of E. Since E is carries the relative topology from R^, 
then o'(C) is closed and bonnded. Now if ^{K) were nonempty, then it would contain a 
component of E. But the components of E are nnbounded, so this is impossible, and so 
(t(C), and hence C, are empty. □ 

We now prove the proposition. 

Proof of Proposition 18. We proceed on a case by case basis, with the assnmption that 
d = 2. 

Case 0. We have already established in Lemma 19 that O 2 /H"’" does not contain an 
open compact subset, and since 1) is conjngate to a Lie snbalgebra of the Heisenberg Lie 
algebra. Theorem 1 applies to show that tt is not integrable. 

Case 1. First recall that here n is nontrivial, and all roots are real. Hence Theorem 
1 applies once again, and yields non-integrability whenever 02 /H'^ does not contain 
nontrivial compact open snbsets. Here A is real and 1) = (H,X)]r where A(H) = 1 and 
A(X) = 0. Choose a basis for R^ for which A has Jordan form, and for which X is lower 
triangular. Put Y = A — \{A)L We distinguish several subcases. 

(a) Suppose that H = 0. If O 2 consists of only one layer, then confer the preceding lemma. 
Suppose then that O 2 is the nnion of the non-empty sets f 22 and Ha. Then H 2 7 ^ 0 implies 
X21 7^ 0 , while Ctz ^ implies that for some n, ui = 0 and pz{Xv) 7^ 0 , hence 0:32 7^ 0 . 

Let U be an open iL^-invariant snbset of O 2 that is connll for Lebesgue measure. Then 
n H 2 is connll and iJ^-invariant, hence 1/ fl S 2 is connll for the Plancherel measnre 
class, where S 2 = {[if; 0 ,^ 3 ] : Ua G R}. But in this case the Plancherel measnre class is 
that of Lebesgue measure on each component of S 2 , so in particular, the set 

E = {z eR: [1,0,2;]* e U} 
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is conull for Lebesgue measure. Now choose a sequence z{k) G E such that z{k) —)■ oo and 
put v{k) = [1, 0, sign(a; 2 ia; 32 ) 2 :(/i:)]*,/c = 1,2,3.... We claim that the sequence H'^v{k) 
has no convergent subsequence in O 2 /H"’". 

Suppose that the claim is false. Let v{j) = [1, 0, sign(a;2ia;32);7(j)]* be a subsequence of 
v{k) for which H'^v{j) converges to H^v, for some v G 02- Since v{k) is not convergent 
in S 2 , then v ^ S 2 , and G Hence we may assume that v = [0, ±1, 0]* G S 3 . 

It follows from the definition of the quotient topology that we have Sj,tj G M such that 
—)■ V, that is, 

[±l,X2ltj,sign{x2lX32)z{j) + Xsitj + X2lX32t‘^j/2] —^ [0,±l,0]h 

Thus —)■ 0 while e^^X 2 itj —)■ ±1, and 

(8) (sign(a;2ia;32)2:(j) + X3itj + X2\X32t]li) —^ 0. 

Now —)■ 0 and e^^X 2 itj —)■ ±1 implies that \tj\ -A- +cxd, and —)■ +cxd. But then 

sign(a; 2 ia; 32 ) (sign(a; 2 ia; 32 )^(j) + X 3 itj +X2iX32ty2) —^ +cx) 
contradicting (8). Thus the claim is proved, and U is not compact. 




X 


X 21 0 or X 

X 31 0 



respectively. If 0:21 7 ^ 0 (in the first case) then O 2 = k} 2 ', otherwise, O 2 = fis- In any case 
Lemma 21 applies. 


(c) Suppose that 7 ^ 0, that is, that 

"A 

H= 1 A 

1 A 

Then X has the form 

■ 0 

X = a; 2 i 0 

Tsi 0:32 0 _ 

where b = X21 = X32. If 6 = 0 then O 2 = H 3 and Lemma 21 applies. Suppose then that 
6 7 ^ 0. We have O 2 = U Hs and both are non-empty. 

We assume b = 1 , and write c = X31. We compute for any n G 

gSA+tV^ = [vi,V2 + (s -h t)vi,V3 - 1 - (s -h t)v2 + {ct + (l/ 2 )(s ty)vi]. 
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The argument now proceeds in a manner similar to that of Case 1(a). Let U be an open 
LT-invariant subset of O 2 that is conull for Lebesgue measure. As before we hnd that the 
set 

E = {z eR: [1,0,;^]* G U} 

is conull for Lebesgue measure. Now choose a sequence z{k) G E such that z{k) —)■ 00 
and put v{k) = [1,0, z{k)Y,k = 1,2,3.... We claim that the sequence H'^v{k) has no 
convergent subsequence in O 2 I■ Supposing the claim is false, then as before this leads 
us to the setup: 

where v{j) is a subsequence of v{k) and where v = [0, ±1, 0]* G S 3 . More explicitly this 
means that 

[1, Sj + tj, z{j) + ctj + {l/2){sj + —)■ [0, ±1, 0]. 

Thus 0, and + tj) ±1, and 

(9) {z{j) + ctj + (1/2)(sj + tj)'^) —)■ 0. 

The hrst two relations imply that e^^^{sj + tj)^ —)■ + cxd , and moreover (since e^^^Sj —)■ 0) 
that e^^Hj —)■ ±1. But e^^^z{j) > 0 and so e^^^{sj +tj)‘^ —)■ +cxd and e^^Hj —)■ ±1 implies 
that 

(^z(yj) + ctj + (l/2)(sj + tj)'^') —y +00 
contradicting (9). Therefore the set U is not compact. 


Case 2: Here we have two dependent roots Ai and A 2 , both real. One of Vi or V 2 has 
dimension two; assume it is Vi. Then we have a basis for and X G t) with matrix form 


X 


0 

1 0 

0 


By Lemma 19 we may assume that both Ai or A 2 are non-vanishing. Hence we can choose 
a basis for such that t) = (A,X)r where X is as above, and where 


1 


A = 


1 


a 


for some a 7 ^ 0. Here it is easily seen that C 2 = O 2 = {u : Ui 7 ^ 0}, for which we have a 
global, connected and noncompact topological section E 2 = {u : Ui = ±1, V 2 = 0, U 3 G M}. 
The Plancherel measure, when transferred to the section E 2 , is again seen to be equivalent 
to Lebesgue measure with respect to the third coordinate. Hence, the same reasoning as 
in the previous cases shows that 02 /H'^ cannot contain a conull compact open subset. 


Again, non-integrability follows from Theorem 1. 


Case 3 : p = 2, Ai and A 2 are independent over M. Write t) = {A,B)s. where \ 2 {B) = 1, 
A 2 (A) = Ai(il) = 0, and where Ai(A) = c G C. 
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(a) Suppose that 3fJc 7 ^ 0. We have t ^ C x M dehned by U 2 , v^Y = [vi + iv 2 , U 3 ]*, 
and via this realization we can write 



It is clear that O2 = {v Vi + iv2 7 ^ 0,^3 7 ^ 0 } and that a topological section for O2 is 
S = {u : = 1, 1 ^ 3 ! = 1}. This section is compact so O 2 /H is compact. 

(b) Suppose that c = i. Let tt : (^2 —)■ C* be the map 7r(u) = ui + iv2. Then tt is 
continuous and open and = e“(ui + iv 2 )] hence we have the continuous open 

mapping n : O2/H —)■ C*/H = (0, +cxd). If iL is a compact open subset of O2IH then 
Ti{K) is a compact open subset of (0, +cxd), hence empty, so K is empty. 

For non-integrability of tt , we observe that the block-diagonal structure of H allows to 
write 

X i/ = (r2 X SO(2)) X (R X R') . 

It follows that the dual space of G = R^ x iL is the cartesian product of the unitary duals 
of the factors. In particular, projecting a nonempty open compact subset of G onto the 
hrst component would yield a nonempty open compact subset in the dual of R^ x SO(2), 
contradicting Theorem 1.1 of [14]. Hence there are no integrable projections in L^(G). 

Case 4: Here each \j is real and the action of is semi-simple. Two of the roots are 
independent, and we can write t) = (H, H)r where 



By Lemma 19 we may assume that one of a or are nonzero. We can write O 2 = 
r 2 i 2 U r 2 i 3 U 023 as a disjoint union, where 

O12 = {u e C>2 : V1V2 7^ 0}, 

Oi3 = {u e C>2 : T2 = 0, V1V3 7^ 0}, 

O 23 = {v e O 2 : vi = 0 , V 2 V 3 7 ^ 0 }. 

Note that if o; = 0, then O 23 = 0 and if /5 = 0 then O 13 = 0. Topological sections in each 
layer respectively are 

S 12 = {[±1, ilWs] : U 3 e R}, Si 3 = {[±1, 0 ± 1]}, S 23 = {[0, ±1, ±1]}. 

We insert a lemma for this particular case. 

Lemma 22. Suppose that t) has the form above. Then the following are equivalent. 

(i) 02 /H^ is compact. 
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(ii) 02 /H'^ contains a compact open conull subset. 

(iii) At least one of a or (3 is positive. 

Proof, (i) implies (ii) is trivial. To prove (ii) implies (iii), suppose that U is an 
invariant open conull subset of O 2 . Then U fl f 2 i 2 7 ^ 0, and it follows that there is an 
open conull subset Z of M such that {±1} x {±1} x Z C S 12 HU. Suppose that neither 
a nor ft are positive, and hence one is negative. Assume that ft < 0. Put 

v{k) = [1,1, z{k)] 

where z{k) G Z and z{k) —)■ 00 . We claim that no subsequence of {H'^v{k)) is convergent. 
Suppose the claim is false; let Sj, tj G M and v{kj) = v{j) be a subsequence that converges 
to a point v in O 2 . Since z{j) —)■ 00 and S 12 is a topological section for r 2 i 2 , then v ^ f 2 i 2 . 
Hence we may assume that n G S 13 U S 23 . Now we have 

e"^"^x(j)] -)■ v. 

with V G Si 3 U S 23 . If n G S 13 , then v = [1,0, ±1]*. Hence e^^ —)■ 1 and eh —)■ 0, 
and so -|-(X) hence z{j) —)■ + cxd , contradicting the convergence above. 

If n G S 23 , then v = [0, l,±l]h Hence —)■ 0 and eb —)■ 1 . Now if a < 0 then 
^asj+ 0 tj while if q; = 0 , then i. In either case z{j) —)■ + cxd , 

contradicting the convergence above. If /9 = 0 then a < 0 and the proof is similar. Thus 
(ii) implies (iii). 

Now suppose (iii). Let v{k) be any sequence in O 2 . In order to show convergence of a 
subsequence of orbits, we may hrst assume that each v{k) belongs to one of the three 
sections above, and then by passing to a subsequence, that every v{k) belongs to one 
of these sections. Now if v{k) G S 13 for all k, or v{k) G S 23 for all k, then since these 
are hnite sets, it is obvious there is a convergent subsequence. Suppose then that v{k) 
belongs to S 12 for all k. 

Thus v{k) = [±1, ±l,n 3 (/c)] for each k. If the sequence Vsi^k) of real numbers is bounded, 
then again it is clear that we have a convergent subsequence, so, passing to a subsequence, 
we may assume that v{k) +cxd or v{k) — 00 . Now suppose that ft > 0. Then we can 
choose tfc G M, /c = 1, 2, 3,... so that 

e^*W 3 (A;) -)■ ± 1 . 

Then —)■ 0 and so 

e*'=^n(A;) ^ [± 1 , 0 ,± 1 ]. 

Similarly, if a > 0, then we can choose s*, G M, /c = 1, 2, 3,... so that 

e^'‘\{k) [0,±1,±1]. 

This completes the proof. □ 

Having obtained the preceding lemma, the characterization of the cases admitting open 
compact conull subsets of the dual orbit space is complete. Furthermore, we already 
obtained a topological section for the case (a), and the hnite open orbit cases (c) through 
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(e) are also taken care of. Hence it remains to take a closer look at the existence of 
quasi-sections for case (b). 

If a = 0, one easily verifies that the set E = {n G : Ui = 1, -1 -t| = 1} is a topological 

section; and the case /9 = 0 is similar. 

Hence it remains to consider the case af3 ^ 0; w.l.o.g. a > 0. Our assumptions imply 
that 

C >2 = {t e : I Til 7 ^ 0 for at least two indices i} . 

Hence, letting for i = 1, 2, 3 and p > 1 

Ciio) = {n e : |Ti| < g ,Vj ^ i - < \vj\ < p} , 

dehnes a compact set Ci{g) C 02- It is therefore sufficient, in view of Lemma 8, to prove 
that ((Oi(p), Oa;(p))) is not relatively compact, for some pair {i,k) of indices. 

First assume that /3 > 0, and consider ((Oi(p), 02(p))). We then need to determine the 
set of pairs {s,t) G such that, with suitable v G the following inequalities hold: 

l^'il < g : ^ < \v2\ <g, ^ < ItsI < g 

i<e*|Ti|<p, e*|T2| < P , J < p . 

These inequalities are solvable with suitable v iff 

-21og(p) <s,t< 21og(p) , -21og(p) <as + f3t< 21og(p) . 

Given any s > 0, we may pick t = —as/(3 < 0 < 2 log(p) so that as -1- /Sf = 0. Hence all 
three inequalities are satished, which means that there exists v G Ci{g) such that exp(sH-|- 
tB)v G G 2 (p), i.e. exp{sA + tB) G ((Gi(p), G2(p))). In particular, ((Gi(p), G2(p))) is not 
relatively compact. 

For the case /5 < 0, we hrst note that a similar reasoning as above leads to the following 
necessary and sufficient conditions for exp(sH -|- tB) to be in ((G 2 (p), G 3 (p))): 

-2 log(p) < s < 2 log(p) , -2 log(p) <t , as + f3t<2 log(p) . 

This can be fulhlled by picking s = 0 and t > 0 arbitrary, showing that ((G 2 (p), G 3 (p))) 
is not relatively compact. 

Again, Theorem 1 applies to yield non-integrability whenever O 2 I contains no nonempty 
open compact set. □ 

5. Combining diagonal and nilpotent generators 

In this section we consider a particular class of abelian dilation groups. Recall that a 
one-parameter group inhnitesimally generated by a diagonal matrix with strictly positive 
entries allows integrable admissible vectors. In this section we investigate what happens 
when we include a second inhnitesimal generator, assumed to be nilpotent. The following 
result shows that this destroys integrability when n > 2. 
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Proposition 23. Let H = exp[), where f) is the real span of two non-zero commuting 
endomorphisms of ML, one of which is nilpotent, and the other diagonalizable with real 
eigenvalues. Suppose that n >3, and let U be an H-invariant, open, conull subset of O 2 . 
Then U/is not compact. It follows that there are no integrable admissible vectors for 

H. 

Observe that if n = 2 and H is as in the proposition, then [) is the real span of X and A 
where 



In this case O 2 = \ : Ti = 0} is the nnion of two open orbits, and O 2 /H is compact. 

Recall that H acts on M"- by h ■ .^ = {h^)~^f, when M"- is identihed with M”. We make 
this identihcation and describe t) in terms of the transposes of its elements. Fix A G I) 
diagonalizable with real eigenvalnes, and write M” = where £ is the set of distinct 

eigenvalues of A, and W\ is the eigenspace for the eigenvalue A. 

Given an eigenspace IF, put IF = ©{IFx : IFx 7 ^ IF}, and let be the projection of 
V G M”. Fix a non-zero nilpotent element X G 1)^. Each IFa is X-invariant, and we 
choose a basis {e[^\ ..., for IFx consisting of A-eigenvectors of A, and for which the 
restriction of X to IFa is of the form 

"0 ■■■O' 

€2 0 ■ ■ ■ 0 
= £3 0 ■■■ 0 

0 . 

and Cj = 0 or 1 . For each X E £ and 1 < i < ra let p^^^ : M” —)■ M denote the coordinate 
function dual to the basis element e\^\ When IF = IFa is hxed we denote p[^^ by p*. The 
following shows that we can assume that all eigenvalues are non-zero. 

Lemma 24. Suppose that A has 0 as an eigenvalue. Then M'^/H'^ contains no nonempty 
compact open subset. 

Proof. Let T = IFo and Ti the span of e^\ ..., e® in T. Let tt : M"" —)■ T/Ti be the 
canonical map dehned by 7r{v) = + Ti. Then tt is continuous and open. Now Ti is H- 

invariant, and the natural action of iL on T/Ti dehned by exp(sA+tX) 7 r(n) = 7 r(exp(s 2 l+ 
tX)v) is trivial. Thus n{Hv) = Hn{v) = 7i{v) is a well-dehned map n : M"’/H —)■ T/Ti 
that is continuous and open. Let K C M ^/be compact and open. Then n{K) is a 
compact and open subset of T/Ti ~ M and hence empty. Therefore K is empty. □ 

For the remainder of this section we assume that 0 ^ T. For each eigenspace IF let 
02{W) = {n G : dimlTv^ = 2}. We have the following. 

Corollary 25. One has 


O 2 = UAe^02(IFA). 
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Proof. Let v E 02- Then necessarily Xv 0. Hence for some eigenspace W 
W kerX. Let b = min{2 <i<n\\ pi{Xw) ^ 0}. Then 


( 10 ) 




^5^ 


and 


( 11 ) 


Ps(e*'^+“'ti) = e^‘(pt(v) + tpi(Xv). 


Since A 7 ^ 0 then dim IIw = 2 and v E 02(IV). 




□ 


Fix an eigenspace W = Wx for which W (fi kerX. Recall that we have chosen a basis for 
W for which X\w has the form 


X|vc — 


■Q 

62 0 
63 


O' 

0 

0 


^nx OJ 


and not all e* are zero. Note that pi{Xv) = 0 and 


( 12 ) 


Pi{Xv) = eiPi-i{v), 2 <i<nx- 


A stratihcation of orbits in 02(W) can be dehned as follows. For each v E 02(IV) put 

b(v^) = min{2 < i < nx ■ Pi{Xv) 7 ^ 0}. 

For 2 < b <nx put H;, = {n G : b{v^) = b}, and let R = {2 < i < ua : e* 7 ^ 0}. Then 
for each 1 < 6 < nx, the relation (12) shows that Qb is non-empty if and only ii b E B, 
and the relations (10) and (11) show that fib is included in 02(W). Thus we have 

02{W) = UbeB 

If T G fib, then pb{X^v) = for all p>2, and hence for any t G M, 

Pi{Xe*^v) = 0 for all i < b, and pb{Xe^^v) = pb{Xv) 7 ^ 0 . 

Thus fib is -invariant. A section for the -orbits in fib is given as follows. 

Lemma 26. The set 

Lb = flbfi{vEMP: Pbip) = 0 } 
is a topological section for the orbits in fib- 


Proof. We have 

Pb{e^^v) = pb{v) + tpb{Xv) = pb{v) -h tpb-i{v) 
holds for all v E fib- Let v E fib, and dehne 


Vbiv) 

Pb{Xv)' 
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Then G Lf,. If n G Lf, and also G Lf,, then 

0 = Pb{e^^v) = pb{v) + tpb{Xv) = t Pb{Xv) 
implies t = 0 and = v. 


□ 

Of particular interest is the “minimal layer” Qa, where a = mini?. Note that Oq = {n G 
M” : Pa{Xv) 7 ^ 0} is open and conull in 

Lemma 27. Let b E B. Then Qb is H-invariant, and the set 

Tb = {v E Lb: Pb{Xv) = ± 1 } 

is a topological section for the H-orbits in fib- 

Proof. Let v G Of,. For b' G i?, if 6' < 6 then we have pb'{Xe^^v) = e^^pb'{Xv) = 0, while 
Pb{Xe^^v) = e^^pb{Xv) ^ 0. Hence E fib and Of, is ii-invariant. To see that Sf, is 
a section, choose t G M as above so that G Lf,, and put s = — In |pf,(Xe*'^n)|. Then 
e^A-htx.^ G Lf, and 

|pf,(Xe*^+*^t;| = |e^"pf,(Xe*^n)| = e*|pf,(Xe*^n| = 1. 

so e^A+tx.^ ^ BEb = fib- If n G Sf, and G Sf, also, then the preceding 

calculations show that t = s = 0. 

□ 

Note that by Corollary 25, each v E O2 belongs to some layer Of, C O^iW). On the other 
hand it is possible that O2 coincides with a minimal layer 0^: in this case there is only 
one eigenspace W that is not included in the kernel of X, and for the matrix of X in W, 
only one subdiagonal entry Ca is non-zero. Applying Lemma 27 we obtain the following. 

Corollary 28. Suppose that n > 3 and O2 = O^. Then 02/H^ does not contain a 
nonempty compact open subset. 

Proof. Since is a topological section for O2/H"’", then its components are homeo- 
morphic with the components of O2/H, and any closed and open subset of O2/H is 
homeomorphic with a closed and open subset of Any non-empty closed and open 
subset of Sq must contain a component. Now since hla is the minimal layer then the 
condition pa{Xv) = ±1 implies that v E fla. Thus by Lemma 27 we have 

Ea = {v EVa'. PaiXv) = ± 1 } 

and Sq is the union of two components. Since n > 3, then each component is an un¬ 
bounded affine hyperplane in I 4 . Hence any non-empty closed and open subset of 02 /Lf^ 
is not compact. But O2IH'^ is Hausdorff, so any compact open subset must be a closed 
and open subset that is bounded, hence must be empty. 


□ 
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We are now ready to prove the proposition. 

Proof of Proposition 23. Let U be an open iL-invariant subset of O 2 that is conull for 
Lebesgue measure. Let W be an eigenspace that is not included in kerX and hi = 
be a minimal layer in 02{W) constructed above. Then U (1 Q is open, conull, and H- 
invariant. If O 2 = hi, then by Corollary 28 we are done, so we suppose that O 2 contains 
other layers Qf,. We scale the generator A so that the eigenvalue for the eigenspace W is 
1 . 

Let S = Sa be the topological section for hi, so that hi = {n G M” : pa{Xv) 7 ^ 0} and 
S = {n G 14 : Pa{Xv) = ±l,Pa{v) = 0}. Put = {n G S : pa{Xv) = 1}; by the relation 

(12) we have 

= {v G 14 : Pa-i{v) = l,Pa{v) = 0}. 

Put q = dim IT. We identify S_|_ with x x R x R'^, via the coordinate 

functions pi,.. .pa- 2 ,Pa+i ■ ■ ■ tPui-i and p^ for hP, and the coordinate functions for all 
other W 7 ^ W. Denote the corresponding n — 2 dimensional Lebesgue measure ma -2 x 
iTini-a-i X mi X uiq ou by m. (If a = 2, then ma -2 is point mass measure.) 

Now U nhl_|_ is conull in hl_|_, where hl_|_ = iLS+, and the measure m is equivalent with the 
push forward of Lebesgue measure on hl+ = {n G hi : pa{Xv) > 0}. Hence m(S+\f/) = 0. 
It follows that there is t G G and w G R^, such that 

(S+ n G R : (t, y, ; 2 , w) G S+ C U} 

is mi-conull in R. Hence there is a sequence z{j) G (S fl such that z{j) —)■ +cxd. 

Dehne the sequence v{j) in S_|_ using the coordinates indicated above by 

v{j) = {x, y, z{j),w), j = 1, 2, 3,.... 

Note that all coordinates of v{j) are hxed except Pni(u(j)) = and the elements x,y 
and w determine all other coordinates of v{j) except Pa-i{v{j)) = l,Pa{v{j)) = 0. We 
claim that Hv{j) has no convergent subsequence in O 2 /H■ 

Suppose the contrary, that for some v G O 2 , and for some subsequence Hv{k) of Hv{j), 
Hv{k) —)■ P[v. Since z{k) —)■ +cxd then v{k) is not convergent in S and S is a topological 
section in hi, therefore Hv ^ Q/H"’". By dehnition of the quotient topology on 02 /H'^, 
we have s{k),t{k) G R, /c = 1, 2, 3,... such that 

v'{k) := —)■ v. 

Now T ^ hi means that pa{Xv) = 0 and so 

(13) Pa{Xv'{k)) ^ 0. 

Recall that pa{Xp^^v{k) = pa{Xv{k) and v{k) G Sq, hence \pa{Xe^^^v{k)) \ = 1. Hence 
Pa{Xv'{k)) = Pa{Xe^'^^v{kY = e^^Pa{Xv{k)) = ±6^'“ 

and so (13) implies 


^ 0. 
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For some eigenspace W = Wx and layer C 02{Wx), we have v G hi;,; in fact we may 
assnme that n G S;,. Hence we have 

(14) pb{v) = 0, Pb{Xv) = ±1. 

Snppose that A < 0. Then —)■ oo, but 

e^'''=P2{Xv{k)) = p2{Xv'{k)) -)■ p2{Xv). 

Hence p 2 {Xv{k)) = p 2 {Xv) = 0. Now if 6 = 2, then this would already the desired 
contradiction in light of (14). Otherwise, p^{{X^v{k)) = 0 for all k and p > 1, so now 

e^*'=P 3 (Xn(A;)) = p^{Xv'{k)) Pz{Xv) 

and if 6 = 3 we arrive at a contradiction. Continuing in this way we obtain pb{Xv) = 0 
while also pb{Xv) = ±1, a contradiction. 

Now suppose that A > 0, so that —)■ 0. Since Pb{v) = 0 then 

(15) Pbiy\k)) 0 
and since \pb{Xv) \ = 1, then 

(16) \pb{Xv\k))\^l. 

We show that the preceding relations are not compatible. First, rewrite (15), (16) as 

(17) 6^*'= pfe(e*'=^ v{k)) 0 

and 


(18) ^ 1. 

To compute pb{e^^^v{k)) and pb{Xe^'=^v{k)), note that for each p > 1, pb{XPv{k)) is 
independent of k, and put Cp = pb{X^v{k)). Then 

Pb{Xe^'^^v{k)) = pb{Xv{k) + tkX‘^v{k) + ^tlX^v{k) H-) 

= Cl + C2tk + ^tl + • • • 

and since 6®'= —)■ 0, then the relation (18) shows that 


Cl + C2tk + + 


oo. 


This shows that not all Cp are zero and \tk\ +cxd. Let N 
N > 1 and we have 


max{p : Cp ^ 0}; then 


so 


e 




Cl + C2tk + • • • + 


Cn 

(A'-l)! 


t 


N-l 

k 


1 




Cl Co 

—-1-=-!-••• + 

Af-l ^ +N-2 ^ ^ 


Cn 


ti 


k ‘'k 

= \tk\ e*" 


(iV- 1)! 


Cl + C2tk + • • • + ' 


+ CXD 
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hence 


Similarly we compute that 


e^^\tk\ —> +CXD. 


Pb{e^>‘^v{k)) = Co + ci4 + ^tl + --- + ^ti . 


Cn ,N 


But now the relation (17) gives 




Co Cl Cat 

— -I- - -1- ■ • • -I-— 

+ + iV! 


= e*'' 


Co + citk +-h 


and hence 

a contradiction. 


= e*'=|pf,(e*''^T(/c)) I —)■ 0 


□ 


Acknowledgements 

Brad Currey would like to thank RWTH Aachen University for their hospitality during 
his visit. Hartmut Fiihr would like to thank Dalhousie University for their hospitality, 
and Karlheinz Grochenig for interesting discussions on the topic. 

References 

[1] S.T. Ali, J.-P. Antoine and J.-P. Gazeau. Coherent states, wavelets and their generalizations. 
Springer-Verlag, New York, 2000. 

[2] B.A. Barnes, The role of minimal idempotents in the representation theory of locally compact groups, 
Proc. Edinburgh Math. Soc. (2) 23 (1980), 229-238. 

[3] D. Bernier and K. Taylor, Wavelets from square-integrable representations, SIAM J. Math. Anal. 
27 (1996), 594-608. 

[4] J. Bruna, J. Cufi, H. Fiihr, and M. Miro, Characterizing abelian admissible groups, J. Geom. Anal. 
25,1045-1074. 

[5] B.N. Currey and R.C. Penney, The strueture of the space of eo-adjoint orbits of a completely solvable 
Lie group, Michigan Math. J. 36 (1989), 309-320. 

[6] J. Christensen and G. Olafsson, Coorbit spaces for dual pairs, Appl. Comput. Harmon. Anal. 31 
(2011), 303324. 

[7] J. Dixmier: C*-Algebras. North Holland, Amsterdam,1977. 

[8] Feichtinger, H., Grochenig, K., Banach spaces related to integrable group representations and their 
atomic decompositions. I., J. Funct. Anal. 86, (1989), 307-340. 

[9] G.B. Folland: A Course in abstract harmonic analysis. CRC Press, Boca Raton, 1995. 

[10] H. Fiihr and M. Mayer, Continuous wavelet transforms from semidirect products: Cyclic represen¬ 
tations and Plancherel measure, J. Fourier Anal. Appl. 8 (2002), 375-398. 

[11] H. Fiihr, Abstract harmonic analysis of continuous wavelet transforms. Lecture Notes in Mathemat¬ 
ics 1863, Springer Verlag Heidelberg, 2005. 

[12] H. Fiihr, Ceneralized Calderon conditions and regular orbit spaces, Colloq. Math. 120 (2010), 103- 
126. 



28 


B. CURREY, H. FUHR, K. TAYLOR 


[13] K. Grochenig, Describing Functions: Frames versus Atomic Decompositions, Monatsh. Math. 112 
(1991), 1-41. 

[14] K. Grochenig, E. Kaniuth and K. Taylor, Compact open sets in duals and projections in -algebras 
of certain semi-direct product groups, Math. Proc. Gambridge Philos. Soc. Ill (1992), 545-556. 

[15] E. Kaniuth and K. Taylor, Projections in C*-algebras of nilpotent groups, Manuscripta Math. 65 
(1989), 93-111. 

[16] E. Kaniuth and K. Taylor, Minimal Projections in -algebras and open points in the dual space of 
semi-direct product groups, J. London Math. Soc. (2) 53 (1996), 141-157. 

[17] E. Kaniuth and K. Taylor. Induced representations of locally compact groups. Gambridge Tracts in 
Mathematics, 197. Gambridge University Press, Cambridge (2013). 

[18] R.S. Laugesen, N. Weaver, G. Weiss and E.N. Wilson, Continuous wavelets associated with a general 
class of admissible groups and their characterization. J. Geom. Anal. 12 (2002), 89-102. 

[19] H. Leptin and J. Ludwig. Unitary representation theory of exponential Lie groups, de Gruyter, 
Berlin, 1994. 

[20] R. Lipsman, Orbital parameters for induced and restricted representations, Trans. Amer. Math. Soc. 
313 (1989), No. 2, 433-473. 

[21] R. Mxirenzi. Ondelettes Multidimensionelles et Application a L Analyse D’lmages. These, Universite 
Catholique de Louvain, Louvain-La-Neuve, Belgium, (1990). 

[22] R. Palais, On the existence of slices for actions of non-compact Lie groups, Ann. of Math. (2) 73 
(1961), 295323. 

[23] N.V. Pedersen, On the characters of exponential solvable Lie groups, Ann. Sci. Ecole Norm. Sup. 
(4) 17 (1984), 1-29. 

[24] A. Valette, Minimal projections, integrable representations and property (T), Arch. Math. 43 (1984), 
397-406. 

[25] A. Valette, Projections in full C*-algebras of semisimple Lie groups. Math. Ann. 294 (1992), 277- 
287. 



